Methods for user interface generation and application modification

ABSTRACT

A method of generating a user interface for presentation to a user. The method comprises executing a first application computer program to provide a user interface, executing agent computer program code to interrogate and modify said user interface during execution of said first application computer program, and presenting said modified user interface. The first application computer program may be run on a server, while the modified user interface may be presented to a user at a client connected to said server.

The present invention relates to systems and methods for generating andmodifying user interfaces. More particularly, but not exclusively, theinvention relates to modifying a user interface provided by anapplication running on a server computer, and providing this modifieduser interface to a user using a client computer connected to thatserver computer. The invention further relates to methods for modifyingan application.

Computers are now ubiquitous in modern society. They are used in manyindustries, and many occupations require users to spend large portionsof a working day using various computer based applications. Typically,users are provided with user interfaces to such applications, the userinterfaces being provided by application suppliers. Such user interfacesnecessarily include a variety of different features, many of which willnot be used by some users. Therefore, a user's productivity may beadversely affected given that a relatively complex user interface isused, when in fact a relatively simple interface could be used to carryout the same task more efficiently.

The problem is worsened when a computer user needs to use a plurality ofdifferent applications in order to complete a task allocated to them,each application having a separate user interface. Such a scenario iscommon in many occupations. Switching between the different userinterfaces of the different applications in order to complete a giventask degrades user efficiency yet further. It will often be the casethat different applications are supplied by different vendors andaccordingly their user interfaces have different “look and feel”, thusfurther degrading user efficiency.

For example, in order to process customer enquiries, operators in a callcentre may need to access a customer management application to accesscustomer details, a billing application to access customer accountinformation, and a payment application to process any payment which maybe made by the customer over the telephone, for example by credit card.Working in this manner is inefficient, given that the operator isrequired to switch between applications in order to complete some tasks.Furthermore, a customer will typically remain on the telephone while theoperator uses these different applications, and it is thereforeadvantageous to speed up the processing of enquires, in order to offer ahigher quality customer service.

Various proposals have been made to enhance user efficiency whenmultiple applications need to be used.

The multiple applications can be combined into a single product orproduct suite. While such a proposal provides great increases in userefficiency, it is difficult and expensive to implement. Furthermore,such a combined product or product suite will typically have a differentuser interface from those used previously, therefore meaning that usersneed to be trained in use of the combined product, further increasingcost.

It has alternatively been proposed that the multiple applications can becombined in some way. For example, all requests can be passed to asingle one of the applications, and this application can be adapted toforward requests to an appropriate source application. Such a solutiontypically requires considerable customisation if it is to work in underall circumstances that may routinely arise, making such a solutiondifficult to implement.

International patent application publication number WO2005/062175describes various methods for user interface generation. Although suchmethods can effectively be used to allow combination of various userinterfaces, they are in general terms intended for use with userinterfaces provided in a plurality of files defined in the HypertextMarkup Language (HTML), which are presented to a user via a web browser.

Thus, there is a need for a generally applicable method for modifyingand combining user interfaces so as to generate user interfaces fordisplay to a user.

It is an object of the present invention to obviate or mitigate at leastsome of the problems set out above.

There is provided a method of modifying a source application. The methodcomprises running the source application and modifying computer programcode associated with the source application at runtime. The modifyingcauses a link to be established between the source application and aninterpreter. The interpreter interprets predetermined modificationcomputer program code, the modification computer program code beingconfigured to modify the source application.

Thus, it can be seen that a method is provided which allows desiredmodifications to be specified in a high level language susceptible ofinterpretation by the interpreter. This modification computer programcode can then be used to modify source application behaviour. Thus,there is provided a framework which allows different modifications to bespecified in a relatively straightforward manner, requiring only highlevel interpreted program code to be created, without the need toperform further modification to the source application. This obviatesthe need for details native code modifications.

The method provided is particularly suitable for modifying userinterfaces.

There is also provided a method and apparatus for generating a userinterface for presentation to a user. The method comprises executing afirst application computer program to provide a user interface, andexecuting agent computer program code to modify the user interfaceduring execution of the first application computer program. The modifieduser interface is then presented.

Thus, there is provided a method in which a user interface is modifiedat runtime during execution of an application computer program code.Thus, a source application provided by different vendor can beconveniently modified.

The method can be carried out on a standalone computer, or alternativelyin a client-server environment. Indeed, the first application computerprogram may be executed on a server, and the server may be incommunication with a client. The modified user interface may then beprovided for presentation at the client. In preferred embodiments of theinvention, the agent computer program code is executed at the server.The agent computer program code may provide the modified user interfaceto the client.

Executing the first application computer program may comprise creatingan application process in which the application computer program is toexecute, and creating a first thread within said application process toexecute the computer program. A second thread may be created within theapplication process to execute the agent computer program code. That is,the agent computer program code may run within the same operating systemprocess as the application computer program. Therefore, the applicationcomputer program and the agent may share operating system allocatedresources.

The invention also provides a further method and apparatus forgenerating a user interface. This method comprises reading first programcode defining the user interface, reading second program code definingat least one modification to the user interface, executing the firstprogram ode within an operating system process having an associatedmemory space, and executing the second program code within the sameoperating system process to generate a modified user interface.

Executing the second computer program code may comprise executingcomputer program code configured to cause an interpreter to run withinsaid operating system process, and commands interpreted using theinterpreter may be used to modify the user interface.

The invention also provides a method and apparatus for affectingoperation of a computer program. This method comprises causing furthercomputer program code to execute within a memory space of the computerprogram. The further computer program code causes execution of aninterpreter configured to interpret instructions to affect operation ofthe computer program. The instructions interpreted by the interpretermay include instructions configured to modify a user interface, althoughit will be appreciated that instructions causing other modifications mayalso be interpreted.

The invention yet further provides a method and apparatus for affectingoperation of a user interface associated with a computer program, duringexecution of that computer program. The computer program has anassociated memory space. The method comprises determining a memorylocation within the memory space at which predetermined computer programis stored. At least part of the predetermined program code is replacedwith further predetermined program code, such that said user interfaceassociated with the computer program is modified. The modificationcomputer program code may be added to a memory space of said computerprogram.

The invention further provides a method and apparatus for affectingoperation of a computer program. This method comprises associating aninterpreter with the computer program, and affecting operation of thecomputer program by interpreting commands using the interpreter.

There is also provided a method of affecting the behaviour of a userinterface comprising a plurality of user interface elements. This methodcomprises defining a model of the user interface using a plurality ofcommunicating objects. The objects are preferably defined using anobject orientated programming language. Each of these objects representsone of the user interface elements. Notification of an event based onactivity within the user interface is received at one of the objects,and that notification is processed at that one of the objects, theprocessing causing behaviour of the user interface to be modified.

It will be appreciated that all aspects of the present invention can beimplemented as methods and apparatus. Additionally, the methods providedby the invention may be implemented using appropriately programmedcomputers. Accordingly, aspects of the invention provide suitablecomputer programs as well as data carriers carrying such computerprograms. The term data carrier is intended to include both tangiblemedia as well as communications lines. The invention can also beimplemented in a distributed fashion, and accordingly aspects of thepresent invention also cover suitably configured computer networks.

Embodiments of the present invention will now be described, by way ofexample, with reference to the accompanying drawings, in which:

FIG. 1 is a schematic illustration of a network of computers suitablefor implementing an embodiment of the present invention;

FIG. 2 is a schematic illustration of a computer of FIG. 1 in furtherdetail;

FIG. 3 is a schematic illustration showing components used to implementan embodiment of the present invention using the network of FIG. 1;

FIG. 4 is a schematic illustration showing part of the embodiment ofFIG. 3 in further detail;

FIG. 5 is a schematic illustration showing operation of a launcherapplication in an embodiment of the present invention;

FIG. 6 is a flowchart showing processing carried out by the launcherapplication of FIG. 5;

FIGS. 7A to 7C are schematic illustrations of threads and a memory spaceof an application process showing the effects of processing illustratedin FIG. 6;

FIG. 8 is a schematic illustration of blocks of computer program codeused to affect application behaviour at runtime;

FIGS. 9 to 12 are illustrations of code fragments taken from the blocksof computer program code shown in FIG. 8;

FIG. 13 is a flowchart showing processing carried out by the computerprogram code of FIG. 12;

FIG. 14 is a schematic illustration of data structures used by thecomputer program code shown in FIG. 8;

FIG. 15 is a schematic illustration showing interactions between nativecode and Python code of FIG. 8 in further detail;

FIG. 16 is a class diagram showing Python classes including the classesshown in FIG. 15;

FIG. 17 is a class diagram showing classes used for communication usingthe classes of FIG. 16;

FIG. 18 is a sequence diagram illustrating usage of the classes of FIGS.16 and 17;

FIG. 19 is a schematic illustration of an object model used to representpart of a graphical user interface;

FIG. 20 is a class diagram showing classes used in the model of FIG. 19;

FIGS. 21 and 22 are sequence diagrams showing usage of the classes ofFIG. 19;

FIG. 23 is a schematic illustration of components used to implement anembodiment of the invention;

FIGS. 24A and 24B are screenshots from a source application;

FIGS. 25A to 25E are screenshots from a target application into whichpart of the source application has been incorporated;

FIGS. 26A to 26C are schematic illustrations showing stages of a processfor code-level modification of a user interface used in an embodiment ofthe present invention;

FIG. 27 is a schematic illustration of a process for generatingmodification code used in the process of FIGS. 26A to 26C;

FIG. 28 is a schematic illustration of a hierarchy of user interfaceelements;

FIG. 29 is a schematic illustration of a hierarchy of modificationprofiles associated with the user interface elements of FIG. 28;

FIG. 30 is a schematic illustration of classes used to provide themodification profiles shown in FIG. 28;

FIGS. 31 to 36 are screenshots taken from an application configured togenerate modification profiles such as those illustrated in FIGS. 29 and30;

FIG. 37 is a schematic illustration of an architecture for implementingan embodiment of the invention in which two source applications run on aserver, to generate a user interface for presentation to a user;

FIG. 38 is a schematic illustration of an embodiment of the presentinvention in which a plurality of source applications are run ondifferent servers, to generate a user interface for presentation to auser, and

FIG. 39 as an alternative embodiment of the invention to thatillustrated in FIG. 37.

Referring first to FIG. 1, it can be seen that a server 1 is connectedto the Internet 2. The server 1 is configured to run a plurality ofsource applications. These source applications can be accessed via theInternet 2 using PCs 3, 4. The PCs 3, 4 are provided with means forconnection to the Internet 2. For example the PCs 3, 4, may be connectedto the Internet 2 via a local area network, via ADSL links or via dialup connections using modems. It will be appreciated that although twoPC's 3, 4 are shown in FIG. 1, in general terms any number of PC's canbe used.

FIG. 2 shows the structure of the PC 3 in further detail. It can be seenthat the PC 3 comprises a CPU 5, and random access memory (RAM) 6 whichin use provides volatile storage for both programs and data. The PC 3further comprises non-volatile storage in the form of a hard disk drive7. An input/output (I/O) interface 8 connects input and output devicesto the PC 3. It can be seen that an output device in the form of adisplay screen 9 is connected to the I/O interface 8. It can also beseen that input devices comprising a mouse 10 a and a keyboard 10 b areconnected to the I/O interface 8. The PC 3 further comprises a networkinterface 11 providing means to connect the PC 3 to a local areanetwork. This link to the local area network can indirectly provideaccess to the Internet 2 as described above. The CPU 5, the RAM 6, thehard disk drive 7, the I/O interface 8 and the network interface 11 areall connected together by means of a communications bus 12.

As described above, the server 1 runs and provides access to sourceapplications. The source applications have associated user interfaces,and these user interfaces (or parts of those interfaces) are displayedto users via the PCs 3, 4. In this way, a user of one of the PCs 3, 4can remotely access source applications operating on the server 1 viathe Internet 2.

Referring to FIG. 3, it can be seen that the server 1 communicates withclients in the form of the PCs 3, 4. The server 1 provides two desktopsessions 13, 14. The desktop session 13 runs a first instance 15 of asource application, while the desktop session 14 runs a second instance16 of the same source application. The desktop session 13 is accessedremotely by the PC 3, while the desktop session 14 is accessed remotelyby the PC 4. Thus, users of the PCs 3, 4 are each provided with accessto a discrete instance of the source application, a user of the PC 3being provided with access to the first instance of the sourceapplication 15, while a user of the PC 4 is provided with access to thesecond instance of the source application 16.

The PCs 3, 4, each run a respective web browser application 17, 18 whichis configured to access the user interface of the respective sourceapplication instance 15, 16. The web browser applications 17, 18, eachcommunicate with a respective one of agents 19, 20 provided by thedesktop sessions 13, 14 respectively. The agents 19, 20 communicate withthe respective source application instances 15, 16 and provide userinterface elements to the web browsers 17, 18 as is described in furtherdetail below.

Referring now to FIG. 4, communication between the PC 3 acting as aclient and the server 1 is described in further detail. As previouslydescribed it can be seen that the server 1 provides a desktop session 13within which the source application instance 15 and its associated agent19 run. The PC 3 runs the web browser application 17 as previouslydescribed. The web browser application 17 displays a HTML page 21 whichincludes user interface elements 22 taken from the interface of thesource application instance 15. In order to provide those user interfaceelements, the HTML page 21 is controlled by a browser agent 23comprising JavaScript code which runs within the web browser 17. Thebrowser agent 23 in turn communicates with the agent 19 operating withinthe desktop session 13 on the server 1. Communication between thebrowser agent 23 and the agent 19 uses the remote desktop protocol(RDP). Communication is carried out via a terminal services applicationclient (TSAC) 24 which runs within the web browser application 17.Operation of the components described with reference to FIG. 4 isdescribed in further detail below. In further detail, in a preferredembodiment of the present invention, the TSAC 24 is provided using theMicrosoft ActiveX RPD client. This client is provided as a “cab” fileand is available from Microsoft Corporation. The client is embeddedwithin the HTML page 22 using javascript code as follows:

-   <OBJECT language-“javascript” ID=“MsRdpClient”    CLASSID=“CLSID:9059f30f-4eb1-4bd2-9fdc-36f43a218f4a”    CODEBASE=“msrdp.cab#version=5,1,2600,2180” WIDTH-800 HEIGHT-“200”>-   </OBJECT>

The client provides a rich scripting interface which allows variousproperties to be defined before communication is established between theTSAC 24 and the agent 19. In particular, it should be noted that theclient provides means to define a virtual communications link which isconfigured to provide a communications channel having desiredproperties. In order to use this communications framework, the agent 23uses a Wtsapi32 API which provides appropriate API calls to allowcommunication.

Having outlined operation of the invention with reference to FIG. 4,operation of an embodiment of the invention is now described in furtherdetail, focussing upon operation of components provided within thedesktop session 13. The described embodiment is based upon the MicrosoftWindows@operating system, although it will be appreciated that otheroperating systems can similarly be used.

As schematically illustrated in FIG. 5, a launcher application 25 runson the server 1 and launches both the source application instance 15 andits associated agent 19, both running within the desktop session 13 onthe server 1. The launcher is an application which is stored on anon-volatile storage device associated with the server 1 and can be runeither from the server 1 or from a computer connected to the server 1via the Internet 2 (FIG. 1).

Although the launcher application 25 can take a variety of forms, in oneembodiment of the present invention, the launcher application 25 islaunched from a command line and takes as a parameter a file system pathfor the source application. The launcher application 25 takes as afurther parameter a delay time. The delay time indicating a time forwhich the launcher application 25 should wait after triggering launch ofthe source application 15 before attempting to create the agent 19, asis described below.

For example, the launcher application may be named tcAppLauncher and becalled from the command line as follows:tcAppLauncher-app“D:\AR System2\ruser.exe”-delay1000  (1)

In this case, the launcher application 25 will launch a sourceapplication “aruser.exe”, and wait for 1000 milliseconds beforeinitiating the agent 19.

The launcher application 25 can also be called using a command of thefollowing form:tcAppLauncher-app“D:\ruser.exe”-rt“C:\crt.dll”-delay1000  (2)

In this case, in addition to specifying the source application“aruser.exe”, and the delay, a “-rt” parameter is used to specify thatthe agent 19 to be used is defined by the “tcrt.dll” dynamic linklibrary (dll). It should be noted that when the “-rt” parameter is notused, the launcher application 25 automatically selects a dll whichprovides the agent 19—that is a dll of predetermined name stored at apredetermined file system path.

The launcher application 25 can also be used to apply the agent 19 to asource application which is already running. In such a case the launcherapplication takes as a parameter a process id (pid) for the sourceapplication which is to be modified. That is, the launcher applicationis launched as follows:tcAppLauncher-pid1580  (3)

From the preceding description of the launcher application 25 and theway in which it is run, it can be seen that in general terms, in BackusNaur Form (BNF), the launcher application is called as follows:

  tcAppLauncher -app < app full path> | -pid <pid> [rt <run-time DLLfull path>]  [-delay <miliseconds>]

Operation of the launcher application 25 is first described in generalterms, with a more detailed description being provided below. In cases(1) and (2) above, the launcher application 25 uses a “CreateProcess”windows API call (provided by the kernel32 dll). This call is used torun the source application 15. After waiting for a specified delay time(specified on the command line as described above for example) for allcomponents of the source application 15 to be property loaded intomemory, the launcher application 25 “injects” a DLL providing the agent19 into the created process. The injected DLL is responsible forperforming modifications to the source application 15 at runtime. Itwill be appreciated that in case (3) above there is no need to callCreateProcess or wait for a predetermined delay time, given that thesource application process already exists.

The injection of a DLL providing the agent 19 into the created processassociated with the source application 15 is now described.

FIG. 6 illustrates processing carried out by the launcher application 25where the launcher application is used to launch the source application(i.e. used in accordance with (1) and (2) above). At step S1, thelauncher application is initiated, as described above, and at step 32,the source application instance 15 is created, again as described above.Having created the source application instance 15, the launcherapplication 25 carries out processing to cause the agent 19 to runwithin the operating system process of the source application 15. Thisinvolves creating an additional thread within the operating systemprocess associated within the source application instance 15. Thisprocess is now described with reference to FIG. 6 and FIGS. 7A to 7C.

FIG. 7A shows a memory space and threads associated with the sourceapplication instance 15, after completion of step S2 of FIG. 8. It canbe seen that the source application 15, comprises threads schematicallydenoted 42 and a memory space schematically denoted 43. It can be seenthat the source application instance 15 comprises a single thread Awhich handles processing associated with the source application 15. Thememory space 43 stores program code required for execution of the sourceapplication. This memory comprises a first portion 44 storing codeassociated with the source application 15. The memory comprises a secondportion 45 storing libraries used by the program code stored in thefirst portion of memory 44. In the described embodiment, these librariestake the form of dynamic link libraries (DLLs). It can be seen that twoof these DLLs are shown in FIG. 7A, namely a kernel32.dll library 46providing access to operating system functions, and a user32.dll libraryproviding access to user interface functions.

It should be noted that the launcher application 25 will have a processidentifier (pid) for the source application 15, obtained from the callto CreateProcess which generated the source application process. Usingthis pid, the launcher application 25 can use various functions toperform processing within the memory space of the source application 15.Specifically, referring back to FIG. 6, at step S3, the launcherapplication 25 uses calls a VirtualAllocEx function provided by theWindows API to obtain a memory allocation. Having obtained a memoryallocation at step S3, a string containing the path of the DLL whichprovides the agent 19 is written to the obtained memory at step S4 usinga call to the WriteProcess memory function, again provided by theWindows API.

Having obtained and written to memory associated with the sourceapplication 15, at step S5 an additional thread is created in the sourceapplication process by the launcher application using theCreateRemoteThread function. At step S6 of FIG. 6, the launcherapplication 25 calls a LoadLibrary function provided by the kernel32.dlllibrary within the source application 15. The LoadLibrary functionoperates within the thread created at step S5. The call to LoadLibraryis possible because standard libraries such as kernel32.dll are loadedat standard offsets from an application start address, and LoadLibraryis itself located at a standard offset within the kernel32.dll library.Thus, given knowledge of these two offsets, and knowledge of the startaddress of the source application instance 15, the launcher application25 can make a call to the LoadLibrary function.

FIG. 7B shows the source application 15 after the LoadLibrary functionhas been called. It can be seen that the threads schematically denoted42 now comprise a second thread B, within which the LoadLibrary functioncall runs. This is the new thread created at step S5 of FIG. 6.

The LoadLibrary function call running within the second thread B loads alibrary, which provides the agent 19 which runs alongside the sourceapplication instance 15. More specifically, this LoadLibrary functioncall loads an agent library which provides various functions used inapplication modification operations, as described below. The loading ofthe agent library by the LoadLibrary function is carried out at step S7of FIG. 6. The agent library is launched within a separate thread withinthe source application 15, as shown in FIG. 9C. Here it can be seen thatthe second thread B has terminated having launched a third thread C. Thethird thread C runs functions of the agent library, which allowoperation of the source application 15 to be affected

In FIG. 7C it can be seen that the second portion of memory 45 nowincludes code of the agent library 48. It can also be seen that thememory 43 allocated to the source application 15 further includes code49 providing the interpreter described above. In preferred embodimentsof the present invention, the interpreter is configured to interpretcommands specified in the Python scripting language.

Operation of the agent 19 at a low level is now described in generalterms. The description is presented with reference to FIG. 8, and withreference to an example in which an application's behaviour in responseto a call to a GetMenu function provided by the user32 dll is modified.As will be described below, behaviour in response to calls to otherfunctions can similarly be modified.

FIG. 8 schematically illustrates the way in which the behaviour of theGetMenu function is changed. It can be seen that the application code 44of the source application 15 includes a call 50 to the GetMenu function.The GetMenu function is provided by the user32 DLL 47, and controltherefore passes from the application code 44 to the code provided bythe user32 DLL 47. It can be seen that the user32 DLL 46 provides ablock of code 51 defining the GetMenu function, and it is this codewhich is executed in response to the function call 50. However, giventhat the GetMenu function is to be modified, so as to providenotification of its call to a component configured to modify itsbehaviour, the first instruction of the GetMenu function 51 has beenmodified to be an unconditional jump having as its target predefinedhooking code 52 stored within a dynamically allocated part 53 of thememory 43 associated with the source application 15. As will bedescribed in further detail below, the hooking code 52 interacts withthe code 51 of the GetMenu function 51 so as to properly affectbehaviour of the GetMenu function 51. It can be seen in FIG. 8 that thehooking code 52 makes appropriate calls to the agent DLL 48 to providemodified behaviour, as is described in further detail below.

Operations carried out to set up the code shown in FIG. 8 are nowdescribed.

FIG. 9 shows the first eleven instructions of the GetMenu functionbefore any modification has taken place. That is, it shows the GetMenufunction before it has been modified so as to include a jump instructionhaving as its target the hooking code 52. Each line of code shown inFIG. 9 comprises an address 54 at which an instruction is stored, and ahexadecimal representation 55 of the instruction stored at that address.Each line further comprises an opcode 56 and one or more parameters 57(if appropriate) represented by the hexadecimal representation. Itshould be noted that code extracts as shown in FIGS. 10, 11, and 12 allinclude similar components.

In order to establish code as illustrated in FIG. 8, the first fivebytes of the GetMenu function 51 are modified so as to provide a jumpinstruction. Before making such a modification, the first five bytes ofthe GetMenu function must be saved, so as to preserve operation of theGetMenu function. Referring again to FIG. 9, it can be seen that a firstmov instruction 58 comprises two bytes, a push instruction 59 comprisesone byte while a second mov instruction 60 comprises two bytes. Thus, itthese three instructions 58, 59, 60 which need to be copied before thejump instruction is inserted as the first instruction of the GetMenufunction.

The three instructions 58, 59, 60 are therefore copied from the GetMenufunction 51 to a trampoline block 61 which is stored within the dynamicmemory 53. This trampoline block is shown in FIG. 10, where it can beseen that it comprises the three instructions 58, 59, 60, and anunconditional jump 62 which targets the fourth instruction 63 of theGetMenu Function (FIG. 9). Thus, if the first three instructions 58, 59,60 of the GetMenu function as shown in FIG. 9 where replaced with a jumpinstruction targeting the start of the trampoline block 61 (i.e.targeting address 0311PFA8), behaviour of the GetMenu function wouldremain unchanged.

However, the first three instructions 58, 59, 60 are replaced with ajump instruction targeting the hooking code 52. FIG. 11 shows theGetMenu function after this amendment has been made. It can be seen thatthe GetMenu function has been modified so as to insert a jumpinstruction 64 which targets the hooking code 52.

FIG. 12 illustrates instructions of the hooking code 52. Operation ofthe hooking code 52 is illustrated by a flowchart shown in FIG. 13. Atstep S8 state is saved by pushing all register values onto a stack. Atstep S9, a LogCall function provided by the agent DLL 48 is called. Itis this call which effects modification of behaviour of the GetMenufunction, and its functionality is described in further detail below.After the call to the LogCall function, state is recovered from thestack at step S10, and control passes to the trampoline 61 at step S11.Thereafter, the trampoline code executes, returning control to theGetMenu function.

Referring to FIG. 12, it can be seen that the hooking code 52 comprisestwo push instructions 65 which save register values onto the stack. Thehooking code 52 further comprises three instructions 66 configured toload and store onto the stack parameters needed by the LogCall function.The hooking code includes an instruction 67 which calls the LogCallfunction. Three instructions 68 restore the register state after thecall to LogCall (so as to ensure proper operation of the remainder ofthe GetMenu function). An instruction 69 passes control back to thetrampoline block 61.

Thus, it can be seen that by launching the agent DLL 48 in its ownthread, modifying instructions of the GetMenu function 51 (having copiedinstructions to a trampoline block 61), and writing appropriate hookingcode 52 to within the memory space of the source application process,the launcher application 25 is able to affect behaviour of the GetMenufunction by ensuing that a LogCall function is called each time theGetMenu function is called. This can best be seen in FIG. 8.

The preceding description has been concerned with operation modificationof the GetMenu function 51. It will be appreciated that a plurality offunctions provided by standard libraries such as user32.dll andkernel32.dll and indeed any other DLL can be suitably modified in thisway.

As explained above, the GetMenu function (and any other functionmodified) is modified so as to call a LogCall function which affects itsbehaviour. This LogCall function is now described. The LogCall functionprovided by the agent DLL 48 has a function prototype as follows:

void LogCall (DWORD funcId, PDWORD pFrame)

That is, the function takes as parameters an identifier of the functionfrom which it was called (GetMenu in the example presented above) and astack frame pointer which provides access to a return address for thefunction, as well as parameters used in its call. The LogCall functionis configured to use the function parameter to call a functionconfigured specifically for the function from which it was called. Thatis, referring again to FIG. 8, it can be seen that the LogCall function70 calls an appropriate implementation of the LogCall function providedby a python interpreter 71. That is, a variety of LogCall functions areprovided by the Python interpreter 71, each being appropriate fordifferent functions, as described in further detail below.

It should be noted that in addition to calling an appropriate LogCallfunction provided within the Python interpreter 71, the LogCall function70 provided by the agent DLL also carries out further modification ofthe function of interest (GetMenu in the example above). Specifically,the LogCall function called from within the hooking code 52 modifies thereturn address of the function of interest on the stack, so as to pointto a block of code 73 within the dynamically allocated memory 53. Theblock of code 73 is configured to call a LogReturn function 74 providedby the agent DLL 48, the LogReturn function 74 being configured tofurther affect the behaviour of the function of interest (i.e. GetMenuin the example presented above). Control returns to the originalapplication code 44 from the LogReturn function.

LogReturn has a similar function prototype to that of LogCall presentedabove, and again operates by delegating responsibility for modificationsto appropriate functions provided by the Python interpreter 71. Suchfunctions are described in further detail below.

It has been indicated above that the LogCall function 70 provided by theagent DLL 48 modifies the return address of the function of interest, soas to ensure that the function of interest calls LogReturn beforecontrol returns to the application code 40. It will be appreciated thatin order to allow control to return to the correct point within theapplication code 40, the original return address which is rewritten withthat of the block of code 73 must be saved. In order to allow this tooccur, a data structure is maintained for each thread. Given that allcode within the agent DLL 48 illustrated in FIG. 8 operates within asingle thread, all code can access this data structure, which is adeclared as being local to a particular thread.

FIG. 14 shows a plurality of data structures configured to store returnaddress data, each data structure being local to a particular thread.The data structures of FIG. 14 all have the same structure, which is nowdescribed with reference to a data structure 75.

The data structure 75 takes the form of a stack storing pointers to datastructures 76 storing return address, function ID pairs. When LogCall iscalled, a pointer (also stored as a thread local variable) isincremented, and a new entry in the data structure 75 is createdpointing to the return address and function identifier of the functionwhich called LogCall. When LogReturn wishes to return control to theapplication code 40, it simply obtains the pointer value, and returns tothe appropriate return address. The data structure shown in FIG. 14 isused because it allows nested calls to be handled effectively.Specifically, if LogCall is caused to be called a second time beforecontrol has been returned to the application code 40, it will beappreciated that the pointer associated with the data structure 75 issimply incremented, and a new entry created. In such a case, the firstreturn from the LogReturn function will handle this nested call, anddecrement the pointer so as to ensure that the subsequent return fromLogReturn correctly returns control the application code 40 responsiblefor the first call to the LogCall function.

It has been described above that implementations of both the LogCall andLogReturn functions are provided by the Python interpreter 71. Thesecalls are now described with reference to FIG. 15. It can be seen thatthe LogCall function 70 and the LogReturn function 74 both make use of aC-Python interface 80 allowing communication with the code runningwithin the python interpreter 71. Before a Python function is called byeither LogCall or LogReturn thread management is ensured by acquiringPython GIL (Global Interpreter Lock), so as to ensure that the calledPython function rums without interruption.

The LogCall function 70 calls a LogCall function 81 provided by coderunning within the Python interpreter 71. This is a generic functionwhich is called in response to all calls to the LogCall function 70.Similarly, a generic LogReturn function 82 provided by the Pythoninterpreter 71 is called in response to call calls to the LogReturnfunction 74. The generic LogCall and LogReturn functions 81, 82 providedby the Python interpreter process the function identifier passed to themas a parameter, and use this to select a specific LogCall and LogRetunfunction which provides operational functionality. In FIG. 15, it can beseen that the LogCall function 81 processes the passed functionidentifier to select between LogCall functions 83, 84, 85, while theLogReturn function 82 uses the function identifier to select one ofthree LogReturn functions 86, 87, 88. It can be seen that the specificLogCall and LogReturn functions are grouped in pairs, each associatedwith a particular function causing the call to LogCall or LogReturn.Specifically, it can be seen that the LogCall and LogReturn functions83, 86 are provided in a class 89 associated with the GetMenu function,the LogCall and LogReturn functions 84, 87 are in a class 90 associatedwith the CreateWindow function, while the LogCall and LogReturnfunctions 85, 88 are in a class 91 associated with the DestroyWindowfunctions. In each case, the specific LogCall and LogReturn functionsprovided are configured to ensure that appropriate events take place, asdescribed below.

FIG. 16 shows a hierarchy of Python classes which include the classes89, 90, 91 mentioned with reference to FIG. 15. It can be seen that theclasses 89, 90, 91 which implement specific instances of the LogCall andLogReturn functions are all subclasses of an abstract HookBase class 92.Any calling function with which it is desired to associate a call toLogCall and LogReturn is simply provided with a corresponding class inthe hierarchy of FIG. 16. It should be noted that the HookBase abstractclass 92 provides various general functionality which is inherited byall classes. For example, the HookBase class 92 includesSaveCallParameters and GetCallParameters methods which are used by allsubclasses in manipulating function call parameters.

It can be seen from FIG. 16 that each of the classes 89, 90, 91 has anassociated parameters class 93, 94, 95 parameters class. These classesare populated by methods provided or inherited by the classes 89, 90, 91to properly store parameters associated with functions calling LogCalland LogReturn. It can be seen that the parameter classes 93, 94, 95 areall subclasses of a FunctionParameterBase class 96. TheFunctionParameterBase class 96 is itself a subclass of a Structure class97, the class 97 providing a structure which is configured by itssubclasses to be suitable for storage of appropriate parameters.

It should be noted that the agent DLL 48 runs within a single thread,and typically does not spawn further threads during its operation. It ishowever important to ensure that all code concerned with interfacemodification operates as quickly as possible, so as to return control tothe application code 40. It also be noted that the Python interpreter 71runs only one single native thread for all Python threads, and ensuressynchronisation using the Python GIL.

It should further be noted that while there is one instance of the codeblocks shown in the dynamic memory 53 for each function which is to bemodified, while there is a single instance of the DLL 48 for each sourceapplication.

The preceding description has been concerned with methods for causingparticular functions (LogCall and LogReturn) to be called when apredetermined function is called by the application code 40. It has beenexplained that these generic functions call generic functions providedwithin the Python interpreter 71, which in turn call specific functionsprovided by the Python interpreter 71. It has been explained that thesespecific functions are provided by classes as illustrated in FIG. 16.

The following description is concerned with associating the techniquesdescribed above with particular GUI elements to cause modification, andalso with appropriate configuration methods.

FIGS. 17 and 18 show classes used to associate calls to LogCall andLogReturn with particular GUI elements. In particular, in the describedexample it is desired to ensure that LogCall and LogReturn are calledwhen a CreateWindow function associated with a particular dynamic windowwithin a GUI is called. In FIG. 17, it can be seen that the class 90providing specific versions of the LogCall and LogReturn functions forthe CreateWindow function is shown, along with its inheritance from theHookBase class 92. It can further be seen that a class 98 is used tomodel a dynamic window of interest. It can be seen that the class 98 isa subclass of a DiscoveryEventListener class 99 which ensures that theclass 98 can be monitored as desired. The class 99 has as a subclass ofSimpleDiscoveryEventListeaer class 100.

SimpleDiscoveryEventListener is a simple implementation ofDiscoveryEventListener. Simple Discovery Event Listener illustrates thatthere might be other clients of Discovery Service than the DynamicWindowclass. For example, there might be cases where configuration code maywant to listen to API calls for reasons other than those related toevents associated with a user interface.

The DiscoveryEventListener class 99 is used by a DiscoverySeverice class101. The DiscoveryService class provides a service to its clients,whereby the clients can register to be notified of calls to a particularfunction associated with a particular GUI element. It will beappreciated that in providing this service the DiscoveryService class101 makes use of objects which are instances of classes such as theclass 98 representing GUI objects, and objects which are instances ofclasses such as the class 90 representing function calls of interest.

It should be noted that the class 90 is additionally a subclass of aDiscoveryAgent class 102 so as to ensure that it provides the necessaryfunctionality.

Use of the classes illustrated in FIG. 17 to ensure that a particularclient is informed of particular manipulations of a particular GUIelement is now described with reference to FIG. 18. Here it can be seenthat a client 98 a (which is an object which is an instance of the class98) calls an init method associated with a DiscoveryService object 101 awhich is an instance of the DiscoveryService class 101. In response tothe this init call, the DiscoveryService object 101 a calls init methodsprovided by objects associated with two functions of interest associatedwith the GUI element represented by object 98 a. It can be seen thesefunctions are the CreateWindow and DestroyWindow functions, which arerespectively represented by objects 90 a and 91 a which are instances ofthe classes 90, 91 shown in FIG. 16 respectively.

When the client 98 a wishes to start causing LogCall and LogReturn to becalled in response to calls to the CreateWindow and DestroyWindowfunctions, a start method provided by the DiscoveryService object 101 ais called, causing install methods to be called on each of the objects90 a, 91 a. These calls to the install methods cause code associatedwith the CreateWindow and DestroyWindow functions to be modified so asto call LogCall and LogReturn, in the manner described above withreference to FIG. 8 and subsequent Figures.

Having made appropriate modifications to the code of the functions to bemodified, the client 98 a then uses a subscribe method provided by theDiscoveryService object 101 a to ensure that calls to LogCall andLogReturn caused by the functions represented by the classes 90 a, 91 aare notified to the client 98 a. At this stage, the client 98 a (whichit will be recalled is an object representing the GUI element ofinterest) will receive notifications when CreateWindow or DestroyWindoware called.

FIG. 18 illustrates a native thread 104 which, at some point during itslifetime calls the CreateWindow function (this call is schematicallynoted A in FIG. 18). Given the operations described above which havebeen carried out, this causes the LogCall function to be called.Initially, within the Python interpreter, this call is directed to theHookBase class before being redirected to the object 90 a. The LogCallfunction then causes stack parameters to be appropriately handled,before using a DiscoveryBventListener object 99 a to inform the client98 a that the CreateWindow function has been called.

Similarly, it can be seen that FIG. 18 shows a call to the DestroyWindowfunction being made (schematically denoted B). This call causes similarprocessing to that described above with reference to the CreateWindowfunction.

The description presented with reference to FIGS. 17 and 18 explains howa client can obtain notification when particular functions are called.It will be appreciated that client objects receiving such notificationswill carry out processing to cause a source application to beappropriately modified. Configuration of objects to receive and act uponnotifications is now described, initially with reference to theschematic illustration of FIG. 19.

FIG. 19 illustrates three objects 106, 107, 108, all of which areinstances of subclasses of a UIObject class which is used to modelobjects of interest within a GUI. It can be seen that an object 106represents a top frame of the GUI, while objects 107, 108 represent openand login dialog boxes respectively. In the illustration of FIG. 19, theobject 106 causes an event to be fired indicating that the top framewhich it represents has been activated. This event is transmitted on amessage bus 109, and passed to the objects 107, 108, both of which areconfigured to listen for such activation events. On being notified ofthe activation event, the objects 107, 108 are configured to causepredetermined processing to be carried out. It can be seen that theobject 107 causes an action defined by an object 110 to be carried out,while the object 108 causes an action defined by an object 111 to becarried out. It can be seen from FIG. 19 that the object 107 is alsoconfigured to carry out an action 112 in response to the top frameobject 106 providing a deactivation event.

The action objects 110, 111, 112 shown in FIG. 19 are configured tocause modification of elements of the GUI. It will be appreciated thatin order to carry out such modification, the action objects 110, 111,112 need a handle to the appropriate user interface element. This isensured by the action of LogCall, and the way in which messages arepassed by the discovery service to the object 106 as described above.

Referring to FIG. 20, classes relevant to the schematic illustration ofFIG. 19 are shown. It can be seen that the dynamic window class 98(shown in FIG. 17) is a subclass of a BaseWindow class 114, as is aStaticWindow class 115. It should be noted that static windowsrepresented by the StaticWindow class 115 exist for the lifetime of anapplication. Such windows can be identified in terms of their handlersrelatively straightforwardly be traversing the window tree of theirapplication.

Dynamic windows represented by the DynamicWindow class 98 are createdand destroyed during an application's lifetime. Accordingly, handlersfor such windows must be dynamically discovered at runtime using methodssuch as those described above. For this reason, the DynamicWindow class98 is a subclass of the DiscoveryEventListener class 100, which is usedby the DiscoveryService class 101, as described above. The StaticWindowclass 115 does not have such relationships.

It can be seen that the BaseWindow class 114 is a subclass of a UIObjectclass 116 which is the class used to model all user interface objects inmodels of the form shown in FIG. 19. It can be seen that the classdiagram of FIG. 20 further shows an Action class 117 which is a subclassof a FunctionObject class 118. The Action class 117 is used to modelactions such as the Actions 110, 111, 112 shown in FIG. 19.

FIG. 20 further shows a Container class 119 which acts as a controllerfor all UIObject objects. The Container class is a subclass of theRuntimeContext class 120.

The Container class 119 makes use of an BventDispatcher class 121 whichallows UIObjects to dispatch events, as described above with referenceto FIG. 19.

The Runtime context class 120 is essentially an interface that theContainer class 119 implements and that interface is used by the clientUIObject to communicate with the Container class 119 and the services.It is part of the contract between the UIObject class 116 and theContainerclass 119.

FIG. 21 is a sequence diagram showing how classes illustrated in FIG. 20interact in the example of FIG. 19. A configuration client 122 calls aninit method provided by a Container object 123, which results in a callbeing made to an init method provided by an EventDispatcher object 124.The configuration client 122 also calls an init method provided by aDiscoveryService object 125. The configuration client then calls asetDiscoveryService method provided by the Container object 123 causingthe Container object 123 to be associated with the DiscoveryServiceobject 125. These method calls effectively set up classes required toallow general message passing operations to take place.

Having carried out the method calls described above, the configurationclient 122 then calls an init method provided by the TopFrame object106, and then uses a setFilterProperties method. The setFilterPropertiesmethod is used to define the way in which the GUI element represented bythe TopFrame object 106 will be identified, and suitable identificationmethods are described in further detail below. Next, the configurationclient 122 calls an init method provided by an Action object 126, andhaving initialised the object 126, the Action object 126 is associatedwith the TopFrame object 106 using an addAction method provided by theTopFrame object 106. Having correctly established the objects 106, 126,the TopFrame object is added to the Container object 122 using anaddUIObject method provided by the Container object 122.

Having correctly established the necessary objects, the TopFrame object106 is activated so as to be ready to fire events as appropriate. Whenappropriate, the TopFrame object 106 fires an event, which is processedby the Container object 122. The Container object 122 calls anotifyEventListners (methods, which in this case causes a call to bemade to the function represented by the Action object 122.

It can be seen from the preceding description that methods are providedwhich allow events associated with static windows to be appropriatelyprocessed. Processing of events associated with dynamic windows is nowdescribed with reference to FIG. 22.

Referring to FIG. 22, it can be seen that the configuration client 122calls an init method provided by the LoginDialog object 108. It is to benoted that because the LoginDialog object 108 represents a dynamicobject, the object 108 may well exist before the corresponding GUIelement is created at runtime. The configuration client 122 calls asetCreationFunction method provided by the LoginDialog object 108 todefine the way in which the LoginDialog object 108 is created. TheSetCreationFunction call specifies an API function to listen to. Thereare number of functions which can be used to create a window, such asCreateWindowA, CreateWindowW, CreateWindoExA, CreateWindowExW and so on.The Login Dialog has to know exactly which function is going to carryout creation, and this is achieved using the SetCreationFunction call.The configuration client 122 then calls an init method to define the wayin which the login dialog is identified, before calling a setCreationpredicate method provided by the LoginDialog object 108 is called.

The LoginDialog object 108 is notified for every call to the creationalfunction (set using SetCreationFunction), at that point a filteringfunction is called which evaluates to True or False on the question “Isthis the created window the one I am looking for?”. SetCreationPredicateis used to set the filtering function. For example:

def filterServiceManagementWindow(rv,params):  winClass =GetClassName(rv)  winText = GetWindowText(rv)  if winClass ==“AfxFrameOrView42” and winText==“Login Status” :   return True  else:  return False

The object 127 shown in FIG. 22 is used to perform filtering as shown inthe example code.

Appropriate action objects 128, 129 to be associated with theLoginDialog object 108 are then created using their init methods. Theseobjects respectively represent actions to be carried out on when theLoginDialog is activated and deactivated. These actions are thenassociated with the LoginDialog object 109, before the Login Dialogobject 108 is added to the Container object 123, before being activatedusing its activate method.

The LoginDialog object 108 then calls getDiscoveryService and subscribemethods. These ensure that the LoginDialog object is informed of eventsof interest occurring within other GUI elements.

FIG. 22 then illustrates processing carried out when LogCall is calledfrom a native thread 130. It can be seen that a call to LogCall is madeto a hook object 131 which correctly handles stack parameters asdescribed above, before calling notifyListeners to notify allappropriate objects, including in this case the LoginDialog object 108.FIG. 22 also shows processing carried out when LogReturn is called. Itcan be seen that this processing is very similar to that for LogCall.

FIG. 23 is a schematic illustration of components of the embodimentdescribed above. It can be seen to show standard DLLs 135 which are usedby the embodiment, as well as an agent DLL 136 which is configuredprovide the interface between application code and code within thePython interpreter. It will be recalled (for example from FIG. 15), thatgeneric LogCall and LogReturn functions are provided within the Pythoninterpreter, and these are represented by Python code 137 in FIG. 23. Itcan be seen that standard Python modules 138 and a Python-operatingsystem interface 139 are also provided, as is a ctypes module 140 whichallows C code to call Python functions as described above.

FIG. 23 further shows framework classes defined in Python (andillustrated in FIGS. 16, 17 and 20) which are used by the invention, anda configuration layer 142 which represents a system configuration, forexample the configuration described with reference to FIG. 19.

It can be seen from FIG. 23 that the Agent DLL 136 and API trace Pythoncode 137 provide an effective interface between DLLS 135 and bespokePython code as provided by the classes 141 and the configuration 142.

The preceding description has presented various components whichtogether allow application user interfaces to be modified at runtime. Anexample of how these components can be used to modify and incorporateuser interface components is now presented.

FIGS. 24A and 24B are screenshots from a helpdesk management application(referred to as the source application), where issues are describedusing the displayed interface. FIG. 24A includes a summary textbox 150.This textbox has an associated button 151 which, when selected, causes adialog 152 illustrated in FIG. 24B to be displayed. The dialog 152provides a larger text box 153 in which summary information can beinput.

FIGS. 25A to 25E show how the interface of FIGS. 24A and 24B can bemodified at runtime, and incorporated into a further user interface(referred to as the target user interface).

Referring first to FIG. 25A, this shows the target user interface. Itcan be seen that a text box 154 is used to enter a problem description,and when an appropriate description has been entered, a submit button155 is used to request an appropriate part of the source applicationinterface together with appropriate data. It should be noted, referringback to FIG. 4, that the interface as displayed in FIG. 25A constitutesthe HTML page 21 of FIG. 4, while the source application of FIG. 24 isthe source application 15 of FIG. 4.

Having selected the submit button 155, an interface as shown in FIG. 25Bis displayed to the user. At this stage, communication between theclient displaying the HTML page 21 and the source application 15 isestablished as described above. The agent 19 then interacts with thesource application 15 to extract a portion of the interface shown inFIG. 24A for display at the client computer. This is shown in FIG. 25Cwhere it can be seen that a portion 156 of the interface of FIG. 24A isnow included within the target interface. Additionally, referring toFIG. 25D, it can be seen that when the button 151 is selected from thetarget interface, the dialog box displayed has been modified.Specifically, its title bar has been removed, and an OK button 157 hasbeen greyed out, so as to prevent changes to the summary being made.Selecting a Submit button 158 from the interface shown in FIG. 25Ccauses the interface of FIG. 25E to be displayed.

FIGS. 24 and 25 therefore show how, using an architecture shown in FIG.4 and techniques such as those described above, a source applicationsinterface can be modified and reused within a target application.

Alternative exemplary embodiments of the present invention are nowdescribed. In particular, embodiments using interpreters to modifyapplication behaviour at runtime are now described. An interpreter canaffect operation of the user interface in a variety of different ways.For example, the interpreter may interpret commands which make calls tofunctions provided by the user32.dll library, these function callsaffecting operation of the user interface. Although such a technique canbe effective, it requires that scripts are written which refer toappropriate elements of the user interface, such that function calls caninclude handlers for the appropriate user interface elements. Techniquesare therefore required to obtain such handlers, and appropriatetechniques are described in further detail below.

In some embodiments of the invention, changes required are implementedby overriding functions provided the user32.dll library associated withthe source application instance 15. When this is done, at run time,modified behaviour is provided by the overridden function in place ofthe usual behaviour of the function provided by the user32.dll library.For example, if it is required that a window of the user interface isgenerated so as to have a different appearance, the CreateWindowfunction may be overridden, so as to provide a modified function whichis called to create the window which is to be created in a modifiedmanner. When carrying out such overriding (as is described below), it isimportant to ensure that the originally provided function is preserved,given that this originally provided function may need to be used forother elements of the user interface. A suitable technique for carryingout such overriding is described with reference to FIGS. 26A to 26C.

FIG. 26A shows the portion of memory 47 which stores the user32.dlllibrary within the memory space of the source application 15. It can beseen that the portion of memory 47 includes program code for a function160 which is to be overridden. FIG. 10 further shows a portion of memory161 which is part of the memory space 43. The portion of memory 161 iscurrently storing no data required by the source application instance15. Although the portions of memory 47, 161 are shown as beingcontiguous in FIG. 26A, it will be appreciated that in practice thisneed not be the case.

As previously described, the start address of the portion of memory 47will be known given that the user32.dll library is always stored withinan application process's memory space at a predetermined offset. Giventhat the agent 19 operates within the source application 15, the startaddress of the memory space of the process will be known. Given thisstart address and knowledge of the predetermined offset, the startaddress of the portion of memory 47 can be computed. Similarly, thestart address for the function 160 can be computed.

The overriding used by the present invention operates by replacing codeat a predetermined address associated with a function with codeconfigured to carry out overriding. Thus when a function call is made atruntime (by redirecting control to the address associated with thefunction), code configured to carry out overriding is executed.

In order to implement the required overriding, it is necessary toreplace a sufficient number of bytes of the function 160 (referred to asthe “first bytes”) with a jump (JMP) instruction, such that when thesource application instance 15 calls the function 160, the JMPinstruction is executed, so as to cause code configured to carry outoverriding to be executed. It will be appreciated that the number ofbytes required to represent a JMP instruction will determine a minimalnumber of first bytes, although the number of first bytes will need tobe selected so as to end at an instruction boundary.

As described above, the first bytes of the function 160 are to beoverwritten by a JMP instruction. In order to preserve operation of thefunction 160, before such overwriting, the first bytes which are to beoverwritten are copied into a portion of memory 162, as shown in FIG.26B. The copy of the first bytes is immediately followed by a JMPinstruction 163 directing control back to the function 160.

Having copied the first bytes of the function 50 into the portion ofmemory 162, the first bytes are replaced by a JMP instruction 164 asshown in FIG. 10C. The JMP instruction 164 has as its target a portionof memory 165 which stores code configured to carry out the overridingdescribed above. This will leave a block of code 166, which is thefunction 160 after removal of the first bytes. Program code within theportion of memory 165 is described in further detail below, although itshould be noted that in general terms this code will check whetheroverriding is appropriate. If overriding is not appropriate, a JMPinstruction will direct execution to the first bytes 162, from where theJMP instruction 163 will direct control back to the code 166, thusproviding functionality of the function 160.

Care must be taken when generating the code 165 configured to carry outthe overriding. In particular, it should be noted that, at runtime,control will be directed to the code 165 immediately after a call to thefunction 160 has been made. Thus, the operating system call stack willcontain parameters for the function 160, and it is thus necessary toensure that the code 165 appropriately interprets these parameters.

A method for generating the code 165 is now described with reference toFIG. 27. In a high level scripting language such as a Python, a functionprototype 167 for the function 160 is created. Additionally, a Pythonfunction 168 which is to be used to generate the code 165 is created.Having created the function 168, standard python techniques are used toassociate the function 168 with the function prototype 167, thusgenerating the code 165.

As indicated above, it is likely that the code 165 will first checkwhether the call made requires the overridden version of the function,and if not, the original version of the function 160 will be used. Ifthis check determines that something more than simply a call to theoriginal function 160 is required this is carried out by the code 165,generated from the Python function 168. The additional operationsrequired can take a wide variety of forms. For example, it may bedesired to suppress operation of the function completely. That is, wherethe function 160 is the CreateWindow function, the code 165 maydetermine the window with which the function call is associated fromparameters on the call stack. A check can then be carried out todetermine whether the operation should be suppressed for that window. Ifthis is the case, the suppression would prevent the window beingdisplayed, thereby causing the user interface to be modified.

It has been described that embodiments of the invention makemodifications to various elements of a user interface. Identification ofthese elements, and modelling of necessary modifications is nowdescribed. FIG. 28 is a schematic illustration of a hierarchy of userinterface elements making up a user interface which is to be modified inan embodiment of the invention. It can be seen that the user interfacecomprises a top level window 170 which comprises two child elements.These child elements are a menu bar 171 and a sub window 172. The menubar 171 has no child elements, while the sub window 172 has two childelements of its own, specifically a menu bar 173 and a further subwindow 174. Again, the menu bar 173 has no child elements while thefurther sub window 174 itself has a single child element in the form ofa menu bar 175.

Details of the hierarchy of user interface elements shown in FIG. 12 canbe obtained from the relevant source application without difficulty.Such details can be obtained either manually or using a computerprogram. FIG. 29 is a schematic illustration of a model used torepresent the user interface shown in FIG. 28, and to model necessarymodifications of that interface. It can be seen that each of the userinterface elements shown in FIG. 28 has a corresponding modificationprofile in the model of FIG. 29. Specifically, a first modificationprofile 176 has two associated child profiles 177, 178. The modificationprofile 176 represents the window 170, while the modification profile177, 178 respectively represent the sub window 172 and the menu bar 171.The modification profile 177 itself has two child modification profiles179, 180 respectively representing the menu bar 173 and the window 174.The modification profile 180 itself has a child modification profile 181representing the menu bar 175.

The modification profiles shown in FIG. 29 represent modifications whichare required to the interface of a source application as describedabove, so as to provide a user interface for display to a user of the PC3. These modification profiles are now discussed in further detail withreference to FIG. 30. FIG. 30 is a class diagram showing how objectorientated classes are used in the modelling process. It can be seenthat a modification profile class 182 has relationships with a detectorclass 183 and an event channel class 184. The modification profile class182 also has relationships with an actions class 185. It can be seenfrom FIG. 30 that events being instances of an event class 186 are senton an event channel represented by an instance of the event channelclass 184.

It will be appreciated that the model represented by FIG. 29 is in factrepresented by a plurality of instances of the modification profileclass 182. Each of these instances of the modification profile class 182will have associated instances of the other classes shown in FIG. 30 asis described in further detail below.

A description of a computer program configured to create and configureinstances of the classes shown in FIG. 30 is now described withreference to the screenshots of FIGS. 31 to 36. The applicationassociated with the screenshots of FIGS. 31 to 36 is run from theinterpreter described above which runs as a thread within a sourceapplication process. Components of the user interface are now described.The user interface comprises a window 190 comprising three panes. Afirst pane 191 represents a hierarchical view of elements of the userinterface of interest. For example, it can be seen that a top level userinterface element 192 comprises a plurality of other user interfaceelements, some of which themselves have child user interface elements. Asecond pane 193 comprises three tabs. A general tab 194 is shown in FIG.31. The general tab 194 displays information such as an object id, titleand handle for a user interface element 195 highlighted in the firstpane 191. Additionally, the general tab 194 includes details of a classname which is used to implement that user interface 3; element. A thirdpane 196 of the window 190 contains details of a modification profileassociated with the user interface element 195 highlighted in the firstpane 191. In the screenshot of FIG. 31, it can be seen that nomodification profile is associated with the highlighted user interfaceelement 195.

Data to be included within the first pane 191, and the second pane 193of the window 190 is obtained by the code running within theinterpreter, and is used to populate the user interface.

Referring now to FIG. 32, a further screen shot of the window 190 isshown. It can be seen here that within the first pane 191 a userinterface element 197 is highlighted. By making appropriate menuselection (for example from a menu displayed when a user clicks a rightmouse button over the user interface element 197) a modification profilefor that object is created, details of which are shown in the third pane196. It can be seen that the modification profile includes threeelements. Specifically, the modification profile includes identificationdata 198, event data 199, and action data 200. The identification data198 indicates data associated with a detector object 183 (FIG. 30)indicating how the user interface associated with the modificationprofile is to be identified.

FIG. 32 shows a situation where no detector object is yet defined.However, by selecting the identification data 198, a suitable detectorobject can be created. Thereafter, when the identification data 198 isselected, the pane 193 displays details of the associated detectorobject. This is shown in FIG. 33. It can be seen that various criteriawhich can be used to identify a user interface element are shown.Specifically, a user interface element can be identified by its class201, its title 202, its parent window class 203, or its parent windowtitle 204. Each of these properties has an associated tick box, andselection of the tick box will cause configuration of the associateddetector object, so as to cause detection to make use of the appropriateparameter. Data shown in the pane 193 in FIG. 33 is stored within anappropriate instance of the detector class as described above.

It will be appreciated that modification of a source application userinterface will need to be carried out at runtime in response toparticular events. For this reason, as shown in FIG. 34, it is possibleto specify event data 199 to be associated with a particular userinterface element. This associates a particular event channel with themodification profile. More specifically, a particular instance of theevent channel class 184 will be associated with the appropriate instanceof the modification profile class 182. Thereafter, events generated andtransmitted on the appropriate event channel instance will be providedto the modification profile. Such events will typically have associatedactions as represented by the actions data 200. These actions willdefine modifications which are required to the user interface data inresponse to the events. The event passing and event handling mechanismused in a preferred embodiment of the invention is described in furtherdetail below.

Referring now to FIGS. 35 and 36, others of the tabs provided in thepane 193 in connection with a particular user interface elementhighlighted in the pane 191 are described. In particular, a styles tab205 is shown in FIG. 35. It can be seen that the styles tab 205 includesa scrollable list of window styles 206 which may be selected by a userusing appropriate tick boxes provided within the scrollable list 206.The scrollable list 206 has three associated buttons. Window styles arespecified within the scrollable list 206, and a button 207 is then beselected so as to apply the selected styles to the window of the userinterface of the source application, such that a user can verify thatthe selected style is that which is required in use. A second button 208is used to clear window styles selected by the user and applied usingthe first button 207 from the user interface of the source application.A third button 209 allows an action to be created so as to generate awindow having the style specified in the scrollable list 206. Havinggenerated such an action, that action can be associated with an event ofthe type described above. Thus, the portion of the styles tab 205described above allows a user to experiment with window styles byselecting styles from the scrollable list 206 and using the first andsecond buttons 207 and 208. When appropriate styles have been selectedto a user's satisfaction, the third button 209 can then be used tocreate an action so as to apply these styles in use when particularevents occur.

The tab 205 further comprises a second scrollable list 210 which hasthree associated buttons 211, 212, 213. The scrollable list 210 allowsextended windows styles to be specified, and operates, along with thebuttons 211, 212, 213 in a manner analogous to that of the scrollablelist 206 and the buttons 207, 208, 209.

FIG. 36 shows a messages tab 214 of the pane 193. The messages tab 214includes an area 215 which details messages which are associated withthe user interface element currently being manipulated. A message withinthe area 215 can be selected, and a set monitor button 216 can then beused so as to monitor all occurrences of such messages. Such monitoringcan be carried out by an appropriate action. A clear button 217 clearsmessage monitoring.

From the preceding discussion, it will be appreciated that the userinterface described with reference to FIGS. 31 to 36 can be used toanalyse a user interface and to associate particular actions withparticular events associated with that user interface. That is, themethods described above can be used so as to trigger particular actionson the occurrence of particular events. Events of interest may occureither within the source application 15, or within the interface 22provided to a user of the client PC 3 within the HTML page 21. Asdescribed above, the agent 19 monitors for occurrences of particularevents and then takes appropriate actions. In order to allow this tohappen, events occur which at the client PC 3 are transmitted by thebrowser agent 23 via the TSAC to the agent 19. Indeed, the agent 19models the client PC 3 as an object, and therefore treats eventsoccurring at the client PC 3 in the same way as events occurring withinthe source application 15. The communications channel establishedbetween the browser agent 23 and the agent 19 (via the TSAC 24) allowsevents to be passed in both directions. That is, the browser agent 23may also process events and take appropriate actions at the client PC 3.Indeed, both the agent 19 and the browser agent 23 listen for events ofinterest (as specified, for example, using the user interface of FIGS.31 to 36) and take appropriate action. In some cases, events detected byeither the agent 19 or the browser agent 23 may simply require an actionin terms of event transmission to the other of the browser agent 23 andthe agent 19.

The preceding discussion has described how the client PC 3 and theserver 1 may communicate so as to provide a user interface taken from asource application running at the server 1, to a user of the client PC3. The description has also described how elements of that userinterface may be modified. However, in some embodiments of the inventionit is desired that an interface comprising user interface elements takenfrom a plurality of source applications are provided to a used of theclient PC 3. A suitable architecture for implementing such an embodimentof the invention is shown in FIG. 37, where like reference numerals areused to refer to components shown also in FIG. 4.

Referring to FIG. 37, it can be seen that within the desktop session 13run on the server 1, in addition to the source application 15, a furthersource application 218 is also run. Typically, the source application219 will be different from the source application 15. The sourceapplication 219 has an associated agent 218, which functions analogouslyto the agent 19 associated with the source application 15. That is, theagent 218 will typically run within a thread within the process of thesource application 219. Additionally, within the desktop session 13, anagent 220 is provided which communicates with the agent 218 associatedwith the source application 219 and the agent 19 associated with thesource application 15. In the embodiment shown in FIG. 37, the agent 220communicates with the browser agent 23 via the TSAC 24. The agent 220runs as a separate process within the desktop session 13 and isconfigured to communicate with the agents 19 and 218 so as to providedata to the browser agent 13 required to generate the interface 22.

Referring now to FIG. 38, an alternative embodiment of the invention isdescribed. Here, it can be seen that again the server 1, and the PC's 3,4 are connected to the Internet 2. However in the embodiment of FIG. 22,a server 221 is also connected to the Internet 2 as is a composer server222. Again, the PC's 3 and 4 are configured so as to provide a userinterface to respective users, the user interface being generated fromelements of source applications provided by appropriate sourceapplication servers. In this case, the interfaces provided by the PC's3, 4 are provided using user interface elements taken from applicationsrunning both on the server 1 and the further server 221. However, itshould be noted that there is no direct communication between the server1 and the PC's 3 and 4, and similarly there is no direct communicationbetween the server 221 and the PC's 3 and 4. Rather, the PC's 3, 4communicate with the composer server 222 which has associatedcomposition data 223. Requests received by the composer server 222 thenresult in user interface elements being requested from sourceapplications running on the server 1, and the server 221. Such userinterface elements are obtained at the server 1 and the server 221 usingagents of the type described above. User interface elements so obtainedare combined by at the composer server 222 so as to generate interfacesfor display to users of the PC's 3, 4.

FIG. 39 shows yet another embodiment of the invention. Here, the clientPC 3 communicates directly with the server 1 and the server 221 via theInternet (not shown). That is, the browser agent 23 is configured tocommunicate directly with the agent 19 associated with the sourceapplication 15 running on the server 1 as well as an agent 224 which isassociated with a source application 225 running on the server 221. Inembodiments of the invention shown in FIG. 39 it will be appreciatedthat the browser agent 23 must be provided with means so as to combineuser interface data received both from the server 1 and the server 221so as to generate a user interface for display to a user.

It will be appreciated that the source applications referred to abovemay take any one of a number of different forms. For example, on theMicrosoft Windows Operating system, applications likely to beencountered include 32-bit third party applications, 16-bit legacyapplications, DOS based applications, emulators, browser basedapplications and Java applications.

32-bit third party applications are in general relatively easy tomanipulate in the manner described above. When such applications areused, user input events are likely to be relatively easy to handle.However user interface events are likely to depend upon the developmenttools used to develop the user interface. That is, knowledge of themanner in which the application was developed is likely to be beneficialwhen handling user interface events associated with such applications.16-bit legacy applications are likely to be harder to manipulate, givendifferent and less accessible event handling mechanisms.

Browser based applications are now reasonably widespread. Suchapplications provide interfaces by way of a series of pages displayed ina web browser. Interfaces provided by means of HTML pages can berelatively easily manipulated using known techniques. The applicant'spublished International Patent Application Publication No. WO2005/062175which is herein incorporated by reference, describes suitabletechniques. When more complex techniques are used to implement userinterfaces, techniques to manipulate code at run-time such as thosedescribed above, may be effectively used. Techniques requiring suchmanipulation include applets, activeX Controls and Macromedia Flashcontent used to provide user interfaces. Java applications may berelatively easily manipulated, given that they are likely to use welldefined windowing toolkits, such as the abstract windowing toolkit (AWT)or Swing.

In addition to the applications described above, it should be noted thatsome applications are generated automatically from predefined data. Suchapplications may include interfaces which have dynamically changingcontrols, or dynamically changing control names, requiring specialprocessing. It should be noted that the modification techniquesoperative at runtime described above can in general be used to affectthe operation of dynamically altered user interface elements.Additionally, some 32-bit applications render all user interfaceelements within the application (that is, they do not use user interfacelibraries or toolkits to obtain user interface elements).

Although a preferred embodiment of the present invention has beendescribed above, it should be noted that the invention can be carriedout in a variety of different ways. For example, when manipulating auser interface of a source application, various techniques can be used.These include using an already exposed application programmer'sinterface (API) which allows the user interface to be manipulated.

Alternatively, where no such API is exposed, an operating systemmessaging layer can be used to expose an API for use. Such an approachrequires a library to be provided for each type of component which is tobe manipulated.

In some embodiments, COM objects used to create the user interface canbe used to enable manipulation of the user interface. Alternativetechniques include the use of analysis of a rendered user interface atpixel level, and use of this analysis to cause appropriate modification.

The modifications to be carried out to the interface of one or moresource applications can include reuse of all or part of a userinterface, disabling and making invisible parts of the user interface,changing operation of user interface elements, combination of userinterfaces provided by different source applications and rearrangementof user interface elements to meet particular usability requirements.The implementation of such modifications will be readily understood byone of ordinary skill in the art.

Embodiments of the invention described above use ActiveState Pythonversion 2.4.1 and version 0.9.6 of the c types module.

Although specific embodiments of the invention have been describedabove, it will be appreciated that various modifications can be made tothe described embodiments, without departing from the spirit and scopeof the present invention. In particular, where references have been madeto particular computer programming languages or operating systems, itwill be appreciated that other programming languages and operatingsystems could be used in alternative embodiments of the invention.

Furthermore, it will be appreciated that although the embodimentsdescribed relate to modification of user interfaces, other modificationsto applications are also contemplated, and fall within the spirit andscope of the invention.

The invention claimed is:
 1. A method of modifying behavior of a userinterface comprising a plurality of user interface elements, the methodcomprising: performing by a computer system programmed with code storedin a memory and executing by a processor of the computer system toconfigure the computer system into a machine for: detecting an eventrepresenting activity with the user interface at a communicating objectrepresenting one of a plurality of communicating objects, wherein amodel of the user interface is defined using the plurality ofcommunicating objects; receiving, via a message bus, a notification ofoccurrence of the detected event that indicates the activity associatedwith the event; processing the notification of the event detected at oneof the plurality of communicating objects to determine a modificationfrom a first behavior of a first function of a first applicationcomputer program associated with the communicating object to a secondbehavior of a second function of the first application computer programin response to the detected event to modify the user interface, whereinprocessing the notification of the event comprises: copying a portion ofthe first function of a first application computer program to bemodified from a first memory location to a second memory location andadding a return call in the second memory location to after the firstmemory location to return to the first application computer programafter modifying behavior of the user interface; replacing the copiedportion of the first application computer program with a function calllocated within the first memory location, wherein the function call isto the second function to modify the user interface with the secondbehavior; executing the function call to the second function to replacethe first function with the second function in response to the detectedevent; executing the second function; modifying the behavior of the userinterface from the first behavior to the second behavior in accordancewith the executed second function; returning to the portion of the firstapplication computer program copied to the second memory location; andexecuting the portion of the first application computer program copiedto the second memory location to return control to the first applicationcomputer program copied to the first memory location.
 2. The method ofclaim 1 wherein receiving, via a message bus, with an object of themodel a notification of occurrence of the detected event furthercomprises: transmitting a message from the one of the objects to atleast one further object in response to the event.
 3. The method ofclaim 1, wherein at least some of the plurality of communicating objectshave associated action objects, the action objects defining modificationbehavior of the model in response to the events.
 4. The method of claim1, further comprising: performing by the computer system programmed withcode stored in the memory and executing by the processor of the computersystem to configure the computer system into the machine further for:generating the user interface using the communicating objects.
 5. Themethod of claim 4 further comprising: performing by the computer systemprogrammed with code stored in the memory and executing by the processorof the computer system to configure the computer system into the machinefurther for: calling a predetermined function with the communicatingobject to generate the notification.
 6. The method according to claim 5,wherein the predetermined function is called when a function associatedwith a predetermined user interface manipulation is called.
 7. Themethod of claim 5, further comprising: performing by the computer systemprogrammed with code stored in the memory and executing by the processorof the computer system to configure the computer system into the machinefurther for: modifying the object executing to provide the userinterface to insert the function call to the predetermined function. 8.The method of claim 1, wherein the objects are implemented in an objectorientated interpreted programming language running within aninterpreter.
 9. The method according to claim 8, further comprising:establishing a link between the user interface and the interpreter. 10.The method according to claim 8, wherein the predetermined function callcalls a function provided by code running in the interpreter.
 11. A datacarrier carrying computer readable program code configured to cause acomputer to carry out a method according to claim 1.